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Abstract 

This  paper  reports  on  the  collaborative 
research  plans  for  the  last  thrust  of  the  Tactical 
Decision  Making  Under  Stress  program 
sponsored  by  the  Office  of  Naval  Research. 
Findings  from  the  research  will  be  used  to 
develop  principles  and  guidelines  for 
combining  training  with  decision  support,  and 
applications  to  advanced  technology 
development  research. 

1  Introduction 

In  its  first  seven  years,  the  Tactical  Decision 
Making  Under  Stress  (TADMUS)  research 
program,  sponsored  by  the  Office  of  Naval 
Research,  was  highly  successful  in  demonstrating 
the  separate  impact  of  training  and  decision 
support  on  combat  team  performance  (Cannon- 
Bowers  &  Salas,  1998).  The  last  two  years  of  the 
TADMUS  program  have  been  focused  on 
conducting  team  research  to  assess  the  combined 
impact  of  training  and  decision  support.  This 
paper  is  an  update  on  the  research  plans,  progress, 
and  lessons  learned  on  the  collaborative  effort 
among  Navy  labs  (Naval  Air  Warfare  Center 
Training  Systems  Division  and  Space  and  Naval 
Warfare  Systems  Center),  Academia  (San  Diego 
State  University),  Industry  (Sonalysts,  Inc.),  and 
fleet  end  users  in  developing  the  plans  for  the  final 
research  thrust. 


2  Background 

Previous  research  has  shown  that  the 
TADMUS  training  interventions  and  the  DSS  have 
improved  performance  under  stress  (Cannon- 
Bowers  &  Salas,  1998;  Kelly  et  al.,  1996).  In  the 
case  of  TADMUS  training  some  experiments  were 
conducted  with  individuals,  but  most  were 
conducted  with  three-  and  five-person  teams.  The 
DSS  research  thrust  involved  evaluating  the 
impact  of  the  DSS  on  the  performance  of  the 
Commanding  Officer  (CO)  and  Tactical  Action 
Officer  (TAO)  dyad.  A  major  task  for  this  last 
experiment  was  to  develop  and  conduct  research, 
whereby,  the  two  major  TADMUS  interventions 
(DSS  and  training)  would  be  linked  together  in  a 
final  demonstration  of  effectiveness  for  tactical 
decision  making  teams.  The  teams  represent  the 
ship’s  air  defense  combat  team,  which  is  composed 
of  the  CO,  TAO,  Air  Warfare  Coordinator 
(AAWC),  Tactical  Information  Coordinator  (TIC), 
Identification  Supervisor  (IDS),  and  Electronic 
Warfare  Supervisor  (EWS).  The  main  hypothesis 
for  the  experiment  is  that  teams  provided  the 
TADMUS  DSS  and  training  will  perform 
significantly  better  than  the  control.  It  is  expected 
that  compared  to  the  control  group,  trained  teams 
using  DSS  will:  a)  be  more  timely  and  accurate  in 
the  Detect-to  Engage  (DTE)  sequence  (i.e.  team 
decision  making),  b)  report  more  accurate  threat 
priorities,  c)  use  more  effective  teamwork 
processes  (information  exchange,  supporting 
behavior,  initiative/  leadership,  and 
communication),  d)  perceive  less  workload  and 
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mental  stress,  and  e)  perceive  greater  utility  of 
DSS  for  making  decisions. 

3  Approach 

The  next  section  outlines  the  experimental 
approach  including  details  on  design,  participants, 
the  task,  research  conditions,  measures,  and 
procedure. 

3.1  Design  and  Participants 

The  experimental  design  is  between  groups 
with  multiple  combat  scenarios  as  post-tests. 
Eight  six-person  teams  performing  on  the  Decision 
Making  Evaluation  Facility  for  Tactical  Teams 
(DEFTT)  simulator  (control  group)  will  be 
compared  with  eight  six-person  teams  provided 
DEFTT,  DSS  and  training  (experimental 
condition).  In  addition,  in  1998,  five  such  teams 
participated  in  a  DSS-only  condition,  without  the 
training  component,  in  order  to  refine  and  validate 
the  experimental  protocol.  Team  members  are 
experienced  Navy  department  head  students  at 
Surface  Warfare  Officer’s  School  (SWOS).  The 
research  protocol  for  the  current  effort  is  based  on 
over  six  years  of  team  research  at  SWOS  and 
other  combat  team  training  facilities  (Johnston, 
Koster,  Black,  &  Seals,  1998;  Johnston,  Poirier,  & 
Smith-Jentsch,  1998;  Kelly  et  al.,  1996). 

3.2  The  Task  and  Research  Conditions 

The  6-person  Combat  Information  Center 
(CIC)  Air  Warfare  (AW)  teams  (TAO,  CO, 
AAWC,  TIC,  IDS,  and  EWS)  perform  the  detect- 
to-engage  (DTE)  sequence  on  four  30-minute 
Persian  Gulf  AW  scenarios.  The  scenarios  are 
event-based  with  time-tags  in  order  to  track 
specific  expected  team  behaviors  throughout  (Oser 
et  al.,  1998).  The  four  teams  in  the  control  group 
condition  perform  their  watchstation  task 
assignments  on  a  PC -based  combat  training  system 
that  simulates  shipboard  CIC  AW  displays 
(DEFTT)  (see  Johnston,  Koster,  Black,  &  Seals, 
1998  and  Johnston,  Poirier,  &  Smith-Jentsch, 
1998  for  details).  In  the  DSS  condition,  the  TAO 


and  CO  were  each  assigned  to  a  DSS,  while  the 
remaining  team  members  worked  on  the  same 
DEFTT  watchstations  as  the  control  group.  Head 
sets  are  worn  to  support  verbal  communications 
among  team  members  and  with  role  players  who 
read  a  script.  The  DEFTT  system  and  the  DSS 
run  in  tandem  with  each  other,  but  the  software 
does  not  communicate.  Any  minor  discrepancies 
in  scenarios  depicted  on  each  system  are  handled 
by  roleplayers.  All  research  participants  have  had 
at  least  48  hours  of  DEFTT  experience  prior  to  the 
experiment. 

3.2.1  Control  Condition:  DEFTT-Only 

In  the  DEFTT-only  condition  the  six  team 
members  use  DEFTT  to  perform  their  individual 
task  assignments.  The  TAO  and  CO  interact  with 
the  AEGIS  Display  Screen  (ADS)  simulation.  The 
TIC,  IDS,  AAWC,  and  EWS  each  have  a 
Command  and  Decision  display.  In  addition,  the 
EWS  has  a  SLQ32  simulated  display.  All 
information  is  unclassified.  The  research  protocol 
for  this  condition  was  based  on  the  typical  combat 
training  simulation  strategy  the  Department  Head 
Students  receive  during  their  six  month  curriculum 
at  SWOS. 

3.2.2  Experimental  Condition:  Combined  DSS 
and  Training 

Decision  Support  System.  In  the 

experimental  condition  the  TAO  and  CO  are 
assigned  the  DSS  (see  Morrison  et  al.,  1998  for 
details).  Prior  to  conducting  scenario  runs  the 
TAO  and  CO  receive  a  45  minute  computer-based 
tutorial  that  describes  display  functions  and  allows 
point  and  click  practice.  For  scenario  runs  DSS 
operates  as  a  standalone,  but  is  synchronized  to 
run  in  tandem  with  DEFTT.  The  same  DEFTT 
displays  are  assigned  to  the  other  team  members  as 
in  the  control  condition.  Any  minor  discrepancies 
in  scenarios  depicted  on  each  system  is  handled  by 
role  players.  The  San  Diego  State  University  eye 
tracking  system  is  used  to  track  TAO  scan  patterns 
on  the  DSS.  Pilot  testing  of  DSS  with  DEFTT 
and  eye  tracking  was  conducted  with  five  teams  in 


1998  in  order  to  refine  the  research  protocol  and 
timing  of  procedures. 

Training  Strategy.  Overall,  it  would  take 
approximately  12  Vi  hours  to  conduct  all  of  the 
TADMUS  training  strategies  together  (see 
Cannon-Bowers  &  Salas,  1998).  It  would  have 
been  an  impossible  and  needless  task  to 
demonstrate  the  combined  effectiveness  of  the 
training  for  the  current  experiment.  Therefore,  the 
training  strategy  for  the  current  effort  was  based 
on  a  need  to  integrate  the  most  effective  training 
innovations  into  an  efficient  program  that  could  be 
implemented  within  the  limited  research  protocol 
established  at  SWOS.  Many  lessons  learned 
during  previous  training  experiments  led  to  the 
decisions  made  to  refine  and  develop  the  current 
training.  Training  priorities  were  established  based 
on  a  literature  review  of  TADMUS  training 
interventions  (McCarthy  et  al.,  1998).  Three  main 
competencies  were  documented  based  on  the 
learning  objectives  identified  in  the  TADMUS 
literature:  a)  Decision  Making  (35  Learning 
Objectives),  b)  Teamwork  (31  Learning 
Objectives),  and  c)  Stress  Management  (11 
Learning  Objectives).  Therefore,  by  way  of 
prioritizing  training  content,  decision  making  and 
teamwork  skills  were  chosen  as  the  primary 
competencies  to  train. 

Secondly,  TADMUS  training  strategies  were 
reviewed  in  terms  of  the  types  of  learning  methods 
that  were  used  (e.g.,  lecture/videotape,  computer 
based  training  (CBT),  scenario-based  training, 
behavior  modeling  and  demonstration, 
performance  feedback).  In  most  cases,  a 
combination  of  these  strategies  were  used. 
Therefore,  a  two-step  approach  for  training 
decision  skills  and  teamwork  skills  competencies 
was  taken.  First,  multimedia  based  training  (CBT 
and  videotape)  was  chosen  as  an  efficient  and 
effective  way  of  teaching  declarative  knowledge  at 
the  individual  level.  Two  training  modules  were 
developed  based  on  this  approach:  Decision 
Making  Skills  Training  and  Team  Dimensional 
Training  (TDT).  The  Decision  Making  Skills 
Training  adapted  training  developed  from  critical 


thinking  research  (Cohen  et  al.,  1998),  as  well  as 
other  research  on  naturalistic  decision  making  and 
training  (Zsambok  &  Klein,  1998).  TDT  was 
developed  and  validated  under  previous  TADMUS 
research,  and  later  refined  under  a  separate  6.3 
research  program  for  shipboard  instructor  training 
and  support  (Smith-Jentsch  et  al.,  1998). 

Scenario-based  team  training  (including  behavioral 
modeling,  performance  feedback,  and  team  self¬ 
correction)  was  developed  based  on  results  of 
research  on  critical  thinking  training,  team  leader 
training,  team  adaptation  and  coordination 
training,  and  team  self-correction  training.  These 
strategies  have  demonstrated  that  individuals  can 
apply  and  practice  previously  learned  declarative 
knowledge  in  a  team  environment  (Cohen  et  al., 
1998;  Serfaty  &  Entin,  1998;  Smith-Jentsch  et  al., 
1996;  Tannenbaum  et  al.,  1998).  Next  is  a 
detailed  description  of  each  of  the  modules. 

Decision  Making  Skills  Training  (Duration  2  Vi 
hours).  Students  participate  in  a  two  and  one  half 
hour  CBT  for  decision  making  skills.  The 
interactive  computer-based  curriculum  is 
comprised  of  seven  modules  using  multiple  CIC 
vignettes  that  have  events  instantiated  with 
workload  and  ambiguity,  team  requirements,  and 
decision  requirements  typical  to  the  CIC. 
Vignettes  are  presented  to  students  in  which  they 
are  asked  to  apply  the  knowledge  they  are 
developing  throughout  the  CBT.  The  CBT 
enables  students  to  understand  and  develop  a 
common  language  on  decision  making  that  they 
can  transfer  to  the  scenario-based  team  training 
environment.  Testing  is  embedded  in  the  training 
to  ensure  students  meet  minimum  knowledge 
requirements.  At  the  end  of  training  students  are 
provided  a  booklet  that  serves  as  a  quick  reference 
study  guide  for  decision  making  skills. 

Team  Dimensional  Training  (Duration  30 
minutes).  Students  receive  a  workshop  comprised 
of  a  10  minute  videotape  on  Teamwork  Skills  and 
Teamwork  Dimensions  and  a  20  minute  instructor 
facilitated  training,  with  the  support  of  CBT,  to 
learn  and  practice  identifying  specific  CIC 


teamwork  behaviors.  Students  receive  a  TDT 
Booklet  to  use  during  the  facilitated  training,  and 
as  a  quick  reference  study  guide  for  teamwork 
skills. 

Scenario-Based  Team  Training  using  DSS 
(Duration  2  Vi  hours).  Immediately  following  the 
TDT  workshop,  students  assemble  in  the  DEFTT 
laboratory  to  begin  scenario-based  team  training. 
The  objective  of  this  training  is  to  enable  students 
to  conduct  structured  scenario  pre-briefings  and 
debriefings  (via  team  self-correction).  The  training 
is  designed  to  enable  members  to  identify  and 
implement  effective  teamwork  process  behaviors 
and  decision  making  skills  during  scenarios.  First, 
the  team  conducts  a  scenario  situation  update  for  a 
TADMUS  scenario  and  then  proceed  to  their 
respective  watch  stations.  During  scenario  run 
observers  record  good  and  poor  examples  of 
teamwork  behaviors  on  the  TDT  Debriefing 
Guide.  In  addition,  observers  evaluate  and  record 
student  performance  on  the  DTE  sequence  for  the 
three  critical  events  in  the  scenario  using  a 
Scenario  Event  Summary  Sheet.  Following 
scenario  run,  the  team  is  provided  the  Scenario 
Event  Summary  Sheet  for  performance  feedback. 

Next,  an  instructor/facilitator  leads  the  team 
through  a  scenario  recap  and  discussion  of  their 
DTE  performance  using  the  DSS  as  an  aid  to 
illustrate  scenario  ground  truth  and  discuss 
decision  making  strategies.  For  example,  Rules  of 
Engagement  call  for  verifying  that  an  inbound 
aircraft  is  clear  of  its  territorial  airspace  before  a 
warning  is  issued.  To  make  matters  even  more 
complicated,  in  littoral  environments  like  the 
Persian  Gulf,  a  potentially  hostile  aircraft  will  often 
be  within  its  weapons  release  range  of  ownship 
before  leaving  its  territorial  airspace.  Situations 
such  as  these  create  substantial  time  pressure,  and 
are  often  mishandled  by  relatively  inexperienced 
students.  The  DSS  permits  the  participants  to 
replay  and  pause  the  scenarios  at  these  crucial 
junctures.  This  feature  facilitates  discussions  that 
help  the  team  plan  how  they  will  handle  similar 
situations  in  the  future.  In  addition,  the  DSS 
debrief  provides  cross  training  by  familiarizing  all 


team  members  with  information  the  TAO  and  CO 
have  available  to  them  on  the  DSS  during  a 
scenario.  Cross  training  research  has  shown  team 
communications  improve  when  members  know 
what  information  needs  to  be  passed  to  other  team 
members  (Blickensderfer  et  al.,  1998). 

Following  DTE  debriefing,  the  last  step  in  the 
process  is  an  instructor  leads  the  team  in  a 
structured  debrief  using  the  TDT  Debriefing 
Guide.  The  instructor  guides  the  students  in  a 
team  self-correction  process  for  identifying 
teamwork  performance  improvements. 

The  second  training  scenario  run  allows  the  team 
to  take  responsibility  for  practicing  and 
demonstrating  the  structured  pre-briefing  and 
debriefing  strategy.  First,  the  team  member 
designated  as  the  CO  conducts  the  structured  pre¬ 
briefing  using  teamwork  performance  objectives 
identified  in  the  previous  debrief.  Next,  the  team 
performs  a  second  TADMUS  scenario  run. 
During  scenario  run,  observers  record  good  and 
poor  examples  of  teamwork  behaviors  on  the  TDT 
Debriefing  Guide.  In  addition,  observers  evaluate 
and  record  DTE  performance  on  the  Scenario 
Event  Summary  Sheet.  Following  scenario  run  the 
team  is  provided  the  Scenario  Event  Summary 
Sheet  as  feedback.  With  minimal  facilitation  from 
the  instructor  the  designated  CO  leads  the  team 
through  a  scenario  recap  and  discussion  of  their 
DTE  performance.  The  CO  directs  the  instructor 
to  manipulate  scenario  recap  on  the  DSS.  Then, 
the  CO  leads  the  team  in  a  structured  debrief  using 
a  blank  TDT  Debriefing  Guide.  The  team 
members  have  to  recall  specific  instances  where 
their  performance  was  good  and  poor,  and  engage 
in  team  self-correction  to  identify  teamwork 
performance  improvements.  Following  these 
activities,  teams  receive  feedback  from  observers 
and  instructors  on  their  debriefing  performance. 

3.3  Measures 

3.3.1  Air  Defense  Warfare  Team  Performance 
Index  (ATPI) 


The  ATPI  provides  a  set  of  scores  on  team 
behaviors  related  to  the  DTE  sequence  (Paris  et 
al.,  1998).  It  provides  diagnostic  information  with 
respect  to  team-level  decision  making  based  on 
Marshall’s  (1995)  theory  and  research  on  decision 
schema  (identification,  elaboration,  planning  and 
execution).  A  paper-based  tool  is  used  by  SMEs 
to  record  whether  or  not,  and  when,  team 
members  performed  event-based  actions  required 
by  a  scenario.  Video  and  audio  tapes  enable 
transcriptions  of  communications  for  further 
analyses. 

A  major  challenge  in  this  research  effort  has  been 
achieving  adequate  rater  agreement  on  the  ATPI 
(Johnston  et  al.,  1997).  Therefore,  significant 
efforts  have  been  directed  at  tool  redesign  and 
rater  training.  To  improve  the  tool’s  sensitivity  it 
was  modified  to  incorporate  a  similar  measurement 
strategy  that  Marshall  et  al.  (1998)  used  in  their 
research  on  CIC  teams.  Marshall’s  research 
showed  that  certain  Team  DTE  behaviors  (e.g., 
level  1  queries,  warnings,  covering,  and 
engagement)  should  be  evaluated  with  respect  to 
both  Planning  and  Execution.  Marshall  determined 
that  team  members  communicate  their  planning, 
but  communication  of  their  execution  of  actions 
may  be  delayed,  and  could  have  an  impact  on 
overall  team  effectiveness.  This  is  an  important 
issue  for  performance  diagnosis  and  subsequent 
remedial  training.  Team  members  should  gain 
knowledge  during  debriefs  on  how  their  behaviors 
had  an  impact  on  the  final  outcome  (to  engage  or 
not  engage). 

Secondly,  rater  training  was  improved.  First,  raters 
were  brought  to  concensus  on  how  they  would 
evaluate  each  stage  of  the  DTE  process.  Then, 
they  developed  concise  and  clear  details  for  the 
rating  criteria.  Last,  a  rater  training  notebook  was 
developed  as  a  quick  reference  guide  so  that  raters 
could  refresh  their  memory  prior  to  evaluating 
transcripts  of  team  communications. 


Periodic  situation  updates  are  requested  by  an 
observer  during  each  scenario  run  to  obtain 
TAO/CO  threat  priorities  and  threat  levels. 
Comparison  of  updates  between  the  control  and 
experimental  conditions  will  be  made  in  order  to 
identify  the  impact  of  the  DSS  and  training  on 
threat  priorities. 

3.3.3  Air  Defense  Warfare  Team  Observation 
Measure  (ATOM) 

ATOM  provides  a  set  of  scores  on  four 
dimensions  of  teamwork  behaviors:  Supporting 
Behavior,  Leadership/Initiative,  Information 
Exchange,  and  Communications  (Johnston  et  al., 
1997).  A  paper-based  tool  is  used  by  trained  raters 
to  record  teamwork  performance.  Video  and  audio 
tape  enable  transcriptions  of  communications  for 
further  analyses.  Comparisons  will  be  made 
between  the  control  and  experimental  conditions 
to  determine  differences  in  teamwork.  In  addition, 
team  behaviors  in  the  DSS-alone  condition  will  be 
compared  to  the  other  two  conditions  to  determine 
the  impact  of  DSS  on  teamwork  behaviors. 

3.3.4  NASA  TLX 

Perceived  workload  and  fatigue  will  be 
measured  with  the  NASA  TLX  (Likert  scale 
version)  to  determine  perceived  stress  differences 
among  conditions  and  across  scenarios. 


3.3.2  TAO/CO  Threat  Prioritization 


3.3.5  Briefing  Protocol  Analyses 

As  a  manipulation  check,  team  communications 
during  scenario  briefings  for  the  control  group  and 
experimental  group  will  be  analyzed  to  determine 
the  impact  of  training  and  DSS  on  briefing 
strategies. 

3.3.6  Eyetracking  Metrics 

Using  the  SMI  EyeLink  System,  the  eye 
movements  of  the  student  assigned  to  the  role  of 
TAO  are  tracked  throughout  DSS  training  and 
DSS  scenario  runs.  Pupil  dilation  and  point-of- 
gaze  data  are  collected  for  both  left  and  right  eyes, 
sampling  at  a  rate  of  250Hz.  The  point  of  gaze 
data  consists  of  the  horizontal  and  vertical 
coordinates  giving  pixel  location  for  each 
observation.  Using  the  procedures  developed  by 
Sandra  Marshall  and  her  research  group,  the  pupil 
dilation  is  analyzed  using  wavelet  theory  to 
produce  estimates  of  cognitive  workload.  These 
estimates  show  how  workload  changes  during  the 
course  of  the  scenario.  The  point-of-gaze  data 
shows  the  usage  of  the  different  parts  of  the  DSS. 
The  overall  percent  of  time  spent  in  every  region 
for  each  scenario  is  tabulated,  as  well  as  the  use  of 
the  regions  during  several  minutes  that  have  been 
identified  as  critical  points  in  the  scenario.  For  the 
critical  events,  the  cognitive  workload  will  be 
correlated  with  the  eye  movements  to  determine 
when  and  where  the  highest  levels  of  mental  effort 
occur. 

3.4  Procedure 

3.4.1  Control  Condition 

The  DEFTT-only  condition  involves  DH 
student  participation  for  a  single  day.  At  the 
beginning  of  the  day  informed  consent  forms  are 
completed  and  team  members  respond  to  a 
demographics  questionnaire  in  order  to  determine 
level  of  expertise.  Team  members  with  the  most 
CIC  expertise  are  assigned  as  TAO  and  CO. 
Following  selection,  team  members  are  trained  on 
their  respective  watch  stations.  A  training 


administrator  provides  an  introduction  to  CIC 
watch  station  responsibilities  and  DEFTT 
functions.  Next,  training  scenarios  are  used  to 
familiarize  DEFTT  operators  with  system 
functions,  operations,  and  team  interactions. 
Information  packets  are  provided  to  members  that 
develop  a  context  and  rationale  for  the  research 
scenarios  (e.g.,  environmentals,  geopolitical, 
situation  update,  ROE,  threat  matrix,  etc.). 
Following  DEFTT  familiarization  training  on  two 
training  scenarios  team  members  participate  in 
four  posttest  scenarios  that  are  counterbalanced 
across  teams.  At  the  end  of  each  scenario  session 
team  members  fill  out  the  NASA  TLX,  and  are 
provided  with  the  Scenario  Event  Summary  Sheet 
to  use  for  review  in  their  debrief.  At  the  end  of 
the  day,  students  are  provided  feedback  on 
performance  after  the  last  scenario  as  a  way  to 
ensure  they  received  training  value  for  their 
efforts. 

3.4.2  Experimental  Condition 

The  experimental  condition  involves  DH 
student  participation  for  two  days.  The  first  day 
requires  students  to  participate  in  the  CBT  for 
decision  skills.  The  second  day  is  similar  to  the 
control  condition  protocol  with  the  addition  of 
DSS,  TDT,  and  scenario-based  team  training. 
While  four  students  learn  DEFTT  functions  the 
TAO  and  CO  students  participate  in  a  45 -minute 
DSS  CBT  tutorial.  Eye  tracking  calibration  takes 
place  during  DSS  practice.  Next,  students  receive 
the  TDT  training  followed  by  the  scenario-based 
team  training  using  the  same  two  training 
scenarios  as  in  the  control  condition.  Then,  team 
members  participate  in  the  four  posttests.  At  the 
end  of  each  scenario  session  team  members  fill  out 
the  NASA  TLX,  and  are  provided  with  the 
Scenario  Event  Summary  Sheet  and  the  TDT 
debriefing  guide  to  use  for  review  in  their  debrief 
along  with  the  DSS.  At  the  end  of  the  day, 
students  are  provided  feedback  on  performance 
after  the  last  scenario  as  a  way  to  ensure  they 
received  training  value  for  their  efforts. 

3.5  Progress  and  Summary 


In  this  paper  we  have  described  the  research 
protocol  and  plans  for  conducting  the  final 
experimental  thrust  for  the  TADMUS  program: 
conducting  a  demonstration  of  the  combined  effect 
of  the  TADMUS  DSS  and  training  on  tactical 
team  performance.  To  date  four  teams  have 
completed  the  experimental  condition  and  four 
teams  have  completed  the  control  condition.  The 
remaining  four  teams  in  the  experimental  and 
control  conditions  will  participate  in  June  1999 
and  September  1999,  respectively.  Including  the 
DSS  only  condition,  the  total  number  of 
participants  to  date  is  78.  Data  analyses  are  in  the 
process  of  being  conducted.  Products  from  this 
effort  will  include: 

♦  Principles  and  guidelines  for  integrating 
training  and  decision  support  systems, 

♦  Designing  diagnostic  tools  for  evaluating  team 
tactical  decision  making  performance, 

♦  Developing  and  conducting  rater  training  and 
concise  documentation  of  rater  guidelines, 

♦  Developing  computer  based  training  for 
decision  skills  and  team  performance, 

♦  Developing  and  Conducting  Scenario-Based 
Training, 

♦  Application  of  gaze  patterns  to  identify 
effective  decision  support  strategies, 

♦  Utilizing  the  DSS  display  capabilities  for 
training  and  debriefing  team  decision  making, 
and 

♦  Efficacy  of  eyetracking  technology  for 
assessing  human  cognition  and  performance 

3.6  Transition  Plans 

A  number  of  significant  transitions  of 
TADMUS  research  methods  are  currently  in 
progress.  Specifically,  the  Science  and  Technology 
Reduced  Manning  Initiative  sponsored  by  the 
Office  of  Naval  Research  is  conducting  a 
demonstration  of  comparing  fully  manned 
shipboard  AW  CIC  teams  to  teams  performing  on 
a  newly  developed  CIC  console  that  will  require 
fewer  AW  team  members — the  Multi-Modal 
Watchstation  (Campbell  et  al.,  1997).  It  is  a  joint 
program  of  academia,  industry  and  four  DoD  labs: 


Naval  Air  Warfare  Center  Training  Systems 
Division,  Space  and  Naval  Warfare  Systems 
Center,  Naval  Surface  Warfare  Center,  and  Naval 
Underwater  Warfare  Center.  The  research 
program  includes  basic,  applied,  and  advanced 
technology  development  initiatives.  The  following 
transitions  are  direct  applications  from  the 
TADMUS  program:  1)  a  modified  ATPI  to  be 
used  as  an  on-line/realtime  measurement  tool,  2) 
the  ATOM  will  be  used  in  its  current  form,  3)  rater 
training  for  the  ATPI,  4)  event-based  scenario 
development  with  two  levels  of  stress,  5)  Decision 
Support  System  display  components,  and  6)  other 
TADMUS  research  protocol  components  (i.e., 
research  team  preparation,  prebriefing  materials, 
role  player  scripts,  etc.). 

In  addition,  the  ATPI  is  being  developed  as  an 
electronic  performance  support  tool  to  aid  the 
reduced  manning  researchers  with  real  time  team 
performance  evaluations.  The  paper-based  device 
will  be  modified  and  applied  to  a  handheld  PC  to 
improve  reliability  and  ease  of  ratings.  Data  input, 
data  reduction,  and  data  presentation  interfaces 
will  be  developed  to  speed  recovery  of  data 
analyses. 
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